Object detection system and method for a work machine using work implement masking

ABSTRACT

A system and method are provided for object detection for a work machine. In an embodiment, data may be received from at least one sensor associated with the work machine and corresponding to a field of view extending from the main frame. Objects are classified in respective locations in the field of view, and at least one segmentation mask is generated corresponding to contours for at least one portion of, or attachment to, the work machine as determined to be in the field of view, wherein each of the at least one segmentation mask defines a respective masked zone in the field of view. One or more of the classified objects may then be determined as separate from the portion/attachment, wherein the at least one segmentation mask is applied to the portion/attachment independently of the one or more separate objects in the field of view. The at least one segmentation mask may be dynamic in nature to account for detected movements of, e.g., attachments to the work machine.

FIELD OF THE DISCLOSURE

The present disclosure relates generally to work machines which includework implements mounted thereon, and to methods of detecting andclassifying objects in a proximity thereof. More particularly, thepresent disclosure relates to object detection methods using imagemasking with respect to work implements mounted to a work machine inorder to separate and favorably distinguish external objects in a fieldof view.

BACKGROUND

Conventional methods are known for detecting objects in the rear of awork machine or equivalent vehicle, but typically are only functionalwhen there are no vehicle components protruding into the field of viewfor associated sensors. As but one example, an automotive rear parkingsensor becomes non-functional when there is a bike rack or trailerattached to the hitch, therefore prompting the driver to turn off thesystem as it cannot differentiate the vehicle attachments from obstaclesbehind the vehicle.

Work machines as the primary subject of the present disclosure may forexample include self-propelled vehicles such as dozers, compact trackloaders, excavator machines, skid steer loaders, and the like whichgrade or otherwise modify the terrain or equivalent working environmentin some way. However, the scope of the present disclosure furtherextends to work machines that are not self-propelled. Such work machinesmay include one or more work implements mounted to a main frame thereof,or otherwise extending from the main frame into an external referencearea which may cause a dangerous condition for objects within a certainproximity of the work machine. When such work implements arerear-mounted, such as for example a ripper, tamping roller, or the like,conventional object detection systems would be subject to the sameinability to differentiate external objects as discussed above for motorvehicle sensors.

BRIEF SUMMARY

The current disclosure provides an enhancement to conventional systems,at least in part by introducing a novel object detection system andmethod for identifying and differentiating objects proximate torear-mounted work implements, without “false positive” detection of thework implements themselves. Such a system may desirably assist operatorsin maintaining situational awareness around the work implement, eventhroughout movement of the work machine and/or of the work implementsrelative to the main frame of the work machine.

A system as disclosed herein may utilize machine learning techniques toidentify the geometry of a work implement and selectively apply anappropriate masking feature, wherein the work implement may beautomatically tracked and dynamically masked for enhanced objectdetection. The system and method may also, or in the alternative,determine which work implements are connected to the machine bycomparing a detected shape thereof to a library of implement shapes,wherein an appropriate mask may be dynamically selected. Work implementsmay include machine-readable components such as for example AprilTagtokens or the like to facilitate the tracking of implement orientationand accordingly dynamic movements of the applied mask.

In one particular embodiment, an object detection method is accordinglydisclosed herein for a work machine comprising one or more workimplements supported from a main frame. The method includes receiving,from at least one sensor (e.g., an image data source) associated withthe work machine, data (e.g., image data) corresponding to a field ofview extending from the main frame. The method further includesclassifying objects in respective locations in the field of view, andgenerating at least one segmentation mask corresponding to contours forat least one portion of, or attachment to, the work machine asdetermined to be in the field of view, wherein each of the at least onesegmentation mask defines a respective masked zone in the field of view.One or more of the classified objects may be determined to be separatefrom the at least one portion of, or attachment to, the work machine,wherein the at least one segmentation mask is applied to the at leastone portion of, or attachment to, the work machine independently of anyof the one or more separate objects in the field of view.

In one exemplary aspect according to the above-referenced embodiment,the method may further include generating images on a display unitcorresponding to the field of view and having the at least one imagesegmentation mask applied thereto.

In another exemplary aspect according to the above-referencedembodiment, the method may further include, during movement of the workmachine, detecting from the received data at least one static portion inthe field of view and one or more dynamic portions relative thereto inthe field of view, and generating the at least one segmentation maskcorresponding to the detected at least one static portion in the fieldof view.

In another exemplary aspect according to the above-referencedembodiment, data corresponding to at least one portion of, and/orattachment to, the work machine may be retrieved and applied forgeneration of an associated segmentation mask based on user input.

In another exemplary aspect according to the above-referencedembodiment, the user input comprises a user selection from among alibrary of selectable portions of and/or attachments to the workmachine.

In another exemplary aspect according to the above-referencedembodiment, the method further includes dynamically generating the atleast one segmentation mask based on determined movements of anattachment to the work machine relative to the field of view.

In another exemplary aspect according to the above-referencedembodiment, the movements of the attachment may be determined based ondetected steering signals for the work machine.

In another exemplary aspect according to the above-referencedembodiment, the movements of the attachment are determined based onfirst input signals from a sensor associated with the attachment andsecond input signals from a sensor associated with the work machine.

In another exemplary aspect according to the above-referencedembodiment, a bounding region for the at least one portion of, orattachment to, the work machine may be determined via imageclassification from the received data.

In another exemplary aspect according to the above-referencedembodiment, a bounding region for the at least one portion of, orattachment to, the work machine may be determined at least in part viaoutput signals from one or more movement sensors associated with therespective portion of, or attachment to, the work machine.

In another exemplary aspect according to the above-referencedembodiment, a bounding region for the at least one portion of, orattachment to, the work machine may be determined by scanning a machinereadable tag associated with the respective at least one portion of, orattachment to, the work machine and retrieving bounding region datacorresponding to the respective at least one portion of, or attachmentto, the work machine from data storage based on the scanned machinereadable tag.

In another exemplary aspect according to the above-referencedembodiment, the method may further include detecting respectivepositions of the one or more further objects determined to be in therespective field of view, relative to the work machine and/or the atleast one portion of, or attachment to, the work machine, andconditionally generating output signals corresponding to at least anidentified unsafe position of an object relative to at least one of thework machine and/or the at least one portion of, or attachment to, thework machine.

In another embodiment, a work machine as disclosed herein may comprise amain frame supported by one or more ground engaging units, one or morework implements supported from the main frame, and at least one sensor(e.g., at least one image data source) configured to generate data(e.g., image data) corresponding to a field of view extending from themain frame. A controller is linked to receive the data from the at leastone sensor, and configured to direct the performance of a methodaccording to the above-referenced embodiment and optionally any one ormore of the associated aspects.

In another embodiment, a non-transitory computer readable medium asdisclosed herein may have a computer program residing thereon andexecutable by a processor to direct the performance of a methodaccording to the above-referenced embodiment and optionally any one ormore of the associated aspects.

Numerous objects, features and advantages of the embodiments set forthherein will be readily apparent to those skilled in the art upon readingof the following disclosure when taken in conjunction with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of a tracked work machine incorporating anembodiment of a work machine and method as disclosed herein.

FIG. 2 is a block diagram representing an exemplary control system forthe work machine according to an embodiment as disclosed herein.

FIGS. 3A-3C are perspective views representing an exemplary rear-viewcamera image, an initial classification of objects in the camera image,and an object separation with an image segmentation mask applied to aportion of the camera image, respectively.

FIG. 4A is a side view representing an exemplary work machine and fieldof view for a camera mounted to a rear portion thereof, including ahuman standing behind a rear-mounted work implement.

FIG. 4B is a perspective view from the work machine of FIG. 4A,representing the human separable by the system from the rear-mountedwork implement in the field of view.

FIG. 5A is a side view representing an exemplary tracked work machineand field of view for a camera mounted to a rear portion thereof,including a human between a rear-mounted work implement and the machinetracks.

FIG. 5B is a perspective view from the work machine of FIG. 5A,representing the human separable by the system from the rear-mountedwork implement in the field of view.

FIG. 6A is a top view representing an exemplary work machine and imagesegmentation mask applied within a field of view for a camera mounted toa rear portion thereof.

FIG. 6B is a top view representing the work machine of FIG. 6A, with theimage segmentation mask being dynamically applied with respect tomovement of the work implement relative to the main frame of the workmachine.

FIG. 7 is a flowchart representing an exemplary embodiment of a methodas disclosed herein.

DETAILED DESCRIPTION

FIG. 1 is a perspective view of a work machine 100. In the illustratedembodiment, the work machine 100 is a crawler dozer having afront-mounted work implement 130 (e.g., ground-engaging blade) and arear-mounted work implement 162 (e.g., ripper), but may include any ofvarious alternative implement configurations (e.g., rear only) or workmachines 100 such as a compact track loader, motor grader, scraper, skidsteer, backhoe, and tractor, to name but a few examples. Whileoperating, the work machine may experience movement in three directionsand rotation in three directions. A direction for the work machine mayalso be referred to with regard to a longitudinal direction 102, alatitudinal or lateral direction 106, and a vertical direction 110.Rotation for work machine 100 may be referred to as roll 104 or the rolldirection, pitch 108 or the pitch direction, and yaw 112 or the yawdirection or heading.

An operator's cab 136 may be located on the main frame 140. Theoperator's cab and a front-mounted working implement 130 may both bemounted on the main frame 140 so that at least in certain embodimentsthe operator's cab faces in the working direction of the workingimplement 130. A control station including a user interface 142 with adisplay unit may be located in the operator's cab 136. As used herein,directions with regard to work machine 100 may be referred to from theperspective of an operator seated within the operator cab 136: the leftof work machine is to the left of such an operator, the right of workmachine is to the right of such an operator, the front or fore of workmachine 100 is the direction such an operator faces, the rear or aft ofwork machine is behind such an operator, the top of work machine isabove such an operator, and the bottom of work machine is below such anoperator.

The term “user interface” 142 as used herein may broadly take the formof a display unit and/or other outputs from the system such as indicatorlights, audible alerts, and the like. The user interface may further oralternatively include various controls or user inputs (e.g., a steeringwheel, joysticks, levers, buttons) for operating the work machine 100,including operation of the engine, hydraulic cylinders, and the like.Such an onboard user interface may be coupled to a vehicle controlsystem via for example a CAN bus arrangement or other equivalent formsof electrical and/or electro-mechanical signal transmission. Anotherform of user interface (not shown) may take the form of a display unit(not shown) that is generated on a remote (i.e., not onboard) computingdevice, which may display outputs such as status indications and/orotherwise enable user interaction such as the providing of inputs to thesystem. In the context of a remote user interface, data transmissionbetween for example the vehicle control system and the user interfacemay take the form of a wireless communications system and associatedcomponents as are conventionally known in the art.

The illustrated work machine 100 further includes a control systemincluding a controller 138 (further described below with respect to FIG.3 ). The controller 138 may be part of the machine control system of thework machine, or it may be a separate control module. Accordingly, thecontroller 138 may generate control signals for controlling theoperation of various actuators throughout the work machine 100, whichmay for example be hydraulic motors, hydraulic piston-cylinder units,electric actuators, or the like. Electronic control signals from thecontroller may for example be received by electro-hydraulic controlvalves associated with respective actuators, wherein theelectro-hydraulic control valves control the flow of hydraulic fluid toand from the respective hydraulic actuators to control the actuationthereof in response to the control signal from the controller.

The controller 138 may include or be functionally linked to the userinterface 142 and optionally be mounted in the operators cab 136 at acontrol panel.

The controller 138 is configured to receive input signals from some orall of various sensors associated with the work machine 100, which mayinclude for example one or more sensors 132 associated with afront-mounted work implement 130, a set of one or more sensors 144affixed to the main frame 140 of the work machine 100 and configured toprovide signals indicative of, e.g., an inclination (slope) of the mainframe or the blade, and a set of one or more sensors 164 affixed to forexample a rear-mounted work implement 162 and configured to providesignals indicative of a relative position thereof. In alternativeembodiments, such sensors 132, 144, 164 may not be affixed directly tothe referenced components but may instead be connected indirectlythrough intermediate components or structures, such as rubberizedmounts. For example, sensor 144 may not be directly affixed to the mainframe 140 but still connected to the frame at a fixed relative positionso as to experience the same motion as the main frame.

The sensor(s) 144 may be configured to provide at least a signalindicative of the inclination of the main frame 140 relative to thedirection of gravity, or to provide a signal or signals indicative ofother positions or velocities of the frame, including its angularposition, velocity, or acceleration in a direction such as the directionof roll 104, pitch 108, yaw 112, or its linear acceleration in alongitudinal 102, latitudinal 106, and/or vertical 110 direction.Sensors may be configured to directly measure inclination, or forexample to measure angular velocity and integrate to arrive atinclination, and may typically, e.g., be comprised of an inertialmeasurement unit (IMU) mounted on the main frame 140 and configured toprovide for example a work machine inclination (slope) signal, orequivalent signals corresponding to the slope of the frame 140, asinputs to the controller 138. Such an IMU 144 may for example be in theform of a three-axis gyroscopic unit configured to detect changes inorientation of the sensor, and thus of the frame 140 to which it isfixed, relative to an initial orientation.

In other embodiments, the sensors may include a plurality of GPS sensingunits fixed relative to the main frame 140 or work implement 130, 162,which can detect the absolute position and orientation of the workmachine 100 or components thereof within an external reference system,and can detect changes in such position and orientation.

An image data source 170 such as for example a stereo camera 170 may becoupled to the work machine 100, for example at an elevated rear portionof the main frame 140 and arranged to provide a field of view 172encompassing at least a rear-mounted work implement 162 and objectsproximate thereto. The image data source 170 is functionally linked tothe controller 138 as further described herein for image processingfeatures and steps.

The controller 138 in an embodiment (not shown) may include or may beassociated with a processor, a computer readable medium, a communicationunit, data storage 178 such as for example a database network, and theaforementioned user interface 142 or control panel having a display. Aninput/output device, such as a keyboard, joystick or other userinterface tool, may be provided so that the human operator may inputinstructions to the controller 138. It is understood that the controllerdescribed herein may be a single controller having all of the describedfunctionality, or it may include multiple controllers wherein thedescribed functionality is distributed among the multiple controllers.

Various operations, steps or algorithms as described in connection withthe controller 138 can be embodied directly in hardware, in a computerprogram product such as a software module executed by a processor, or ina combination of the two. The computer program product can reside in RAMmemory, flash memory, ROM memory, EPROM memory, EEPROM memory,registers, hard disk, a removable disk, or any other form ofcomputer-readable medium known in the art. An exemplarycomputer-readable medium can be coupled to the processor such that theprocessor can read information from, and write information to, thememory/storage medium. In the alternative, the medium can be integral tothe processor. The processor and the medium can reside in an applicationspecific integrated circuit (ASIC). The ASIC can reside in a userterminal. In the alternative, the processor and the medium can reside asdiscrete components in a user terminal.

The term “processor” as used herein may refer to at leastgeneral-purpose or specific-purpose processing devices and/or logic asmay be understood by one of skill in the art, including but not limitedto a microprocessor, a microcontroller, a state machine, and the like. Aprocessor can also be implemented as a combination of computing devices,e.g., a combination of a DSP and a microprocessor, a plurality ofmicroprocessors, one or more microprocessors in conjunction with a DSPcore, or any other such configuration.

The communication unit may support or provide communications between thecontroller 138 and external systems or devices, and/or support orprovide communication interface with respect to internal components ofthe work machine 100. The communications unit may include wirelesscommunication system components (e.g., via cellular modem, WiFi,Bluetooth or the like) and/or may include one or more wiredcommunications terminals such as universal serial bus ports.

Data storage 178 as discussed herein may, unless otherwise stated,generally encompass hardware such as volatile or non-volatile storagedevices, drives, memory, or other storage media, as well as one or moredatabases residing thereon.

The work machine 100 is supported on the ground by an undercarriage 114.The undercarriage 114 includes ground engaging units 116, 118, which inthe present example are formed by a left track 116 and a right track118, and provide tractive force for the work machine 100. Each track maybe comprised of shoes with grousers that sink into the ground toincrease traction, and interconnecting components that allow the tracksto rotate about front idlers 120, track rollers 122, rear sprockets 124and top idlers 126. Such interconnecting components may include links,pins, bushings, and guides, to name a few components. Front idlers 120,track rollers 122, and rear sprockets 124, on both the left and rightsides of the work machine 100, provide support for the work machine 100on the ground. Front idlers 120, track rollers 122, rear sprockets 124,and top idlers 126 are all pivotally connected to the remainder of thework machine 100 and rotationally coupled to their respective tracks soas to rotate with those tracks. The track frame 128 provides structuralsupport or strength to these components and the remainder of theundercarriage 114. In alternative embodiments, the ground engaging units116, 118 may comprise, e.g., wheels on the left and right sides of thework machine.

Front idlers 120 are positioned at the longitudinal front of the lefttrack 116 and the right track 118 and provide a rotating surface for thetracks to rotate about and a support point to transfer force between thework machine 100 and the ground. The left and right tracks 116, 118rotate about the front idlers 120 as they transition between theirvertically lower and vertically upper portions parallel to the ground,so approximately half of the outer diameter of each of the front idlers120 is engaged with the respective left 116 or right track 118. Thisengagement may be through a sprocket and pin arrangement, where pinsincluded in the left 116 and right tracks 118 are engaged by recesses inthe front idler 120 so as to transfer force. This engagement alsoresults in the vertical height of the left and right tracks 116, 118being only slightly larger than the outer diameter of each of the frontidlers 120 at the longitudinal front of the tracks. Forward engagingpoints 130 of the tracks 116, 118 can be approximated as the point oneach track vertically below the center of the front idlers 120, which isthe forward point of the tracks which engages the ground.

Track rollers 122 are longitudinally positioned between the front idler120 and the rear sprocket 124 along the bottom left and bottom rightsides of the work machine 100. Each of the track rollers 122 may berotationally coupled to the left track 116 or the right track 118through engagement between an upper surface of the tracks and a lowersurface of the track rollers 122. This configuration may allow the trackrollers 122 to provide support to the work machine 100, and inparticular may allow for the transfer of forces in the verticaldirection between the work machine and the ground. This configurationalso resists the upward deflection of the left and right tracks 116, 118as they traverse an upward ground feature whose longitudinal length isless than the distance between the front idler 120 and the rear sprocket124.

Rear sprockets 124 may be positioned at the longitudinal rear of each ofthe left track 116 and the right track 118 and, similar to the frontidlers 120, provide a rotating surface for the tracks to rotate aboutand a support point to transfer force between the work machine 100 andthe ground. The left and right tracks 116, 118 rotate about the rearsprockets as they transition between their vertically lower andvertically upper portions parallel to the ground, so approximately halfof the outer diameter of each of the rear sprockets 124 is engaged withthe respective left or right track 116, 118. This engagement may bethrough a sprocket and pin arrangement, where pins included in the leftand right tracks are engaged by recesses in the rear sprockets 124 totransfer force. This engagement also results in the vertical heights ofthe tracks being only slightly larger than the outer diameter of each ofthe rear sprockets 124 at the longitudinal back or rear of therespective track. The rearmost engaging point of the tracks can beapproximated as the point on each track vertically below the center ofthe rear sprockets, which is the rearmost point of the track whichengages the ground. In this embodiment, each of the rear sprockets 124may be powered by a rotationally coupled hydraulic motor so as to drivethe left track 116 and the right track 118 and thereby controlpropulsion and traction for the work machine 100. Each of the left andright hydraulic motors may receive pressurized hydraulic fluid from ahydrostatic pump whose direction of flow and displacement controls thedirection of rotation and speed of rotation for the left and righthydraulic motors. Each hydrostatic pump may be driven by an engine 134(or equivalent power source) of the work machine and may be controlledby an operator in the operator cab 136 issuing commands which may bereceived by the controller 138 and communicated to the left and righthydrostatic pumps. In alternative embodiments, each of the rearsprockets may be driven by a rotationally coupled electric motor or amechanical system transmitting power from the engine.

Top idlers 126 are longitudinally positioned between the front idlers120 and the rear sprockets 124 along the left and right sides of thework machine 100 above the track rollers 122. Similar to the trackrollers, each of the top idlers may be rotationally coupled to the lefttrack 116 or the right track 118 through engagement between a lowersurface of the tracks and an upper surface of the top idlers. Thisconfiguration may allow the top idlers to support the tracks for thelongitudinal span between the front idler and the rear sprocket andprevent downward deflection of the upper portion of the tracks parallelto the ground between the front idler and the rear sprocket.

The blade assembly 130 as represented in the embodiment of FIG. 1 is afront-mounted work implement 130 which may engage the ground ormaterial, for example to move material from one location to another andto create features on the ground, including flat areas, grades, hills,roads, or more complexly shaped features. The blade 130 is movablyconnected to the main frame 140 of the work machine 100 through alinkage 146 which supports and actuates the blade and is configured toallow the blade to be lifted (i.e., raised or lowered in the verticaldirection 110) relative to the main frame. The linkage 146 includes ac-frame 148, a structural member with a C-shape positioned rearward ofthe blade 130, with the C-shape open toward the rear of the work machine100. The blade 130 may be lifted (i.e., raised or lowered) relative tothe work machine 100 by the actuation of lift cylinders 150, which mayraise and lower the c-frame 148. The blade 130 may be tilted relative tothe work machine 100 by the actuation of a tilt cylinder 152, which mayalso be referred to as moving the blade in the direction of roll 104.The blade 130 may be angled relative to the work machine 100 by theactuation of angle cylinders 154, which may also be referred to asmoving the blade in the direction of yaw 112. Each of the lift cylinders150, tilt cylinder 152, and angle cylinders 154 may for example be adouble acting hydraulic cylinder.

The ripper assembly 162 as represented in the embodiment of FIG. 1 is arear-mounted work implement 162 which also may selectively engage theground or material, for example to loosen the ground behind the workmachine 100. The ripper assembly 162 as shown includes a plurality of(e.g., three) separate ripper shanks which are typically substantiallyperpendicular to the ground. When the ripper is not in use, the shanksmay be raised so that they are not in contact with the ground.Alternatively, when the ripper is in use, the shanks may be lowered topenetrate the ground surface and thereby loosen the ground as the workmachine proceeds.

As schematically illustrated in FIG. 2 , the work machine 100 in anembodiment as disclosed herein includes a control system 200 including acontroller 138. The controller 138 may be part of the machine controlsystem of the work machine 100, or it may be a separate control module.The control system 200 may include hydraulic and electrical componentsfor controlling respective positions of the front-mounted 130 and/orrear-mounted 162 work implements. For example with respect to the blade130, each of the lift cylinders 150, the tilt cylinder 152, and theangle cylinders 154 is hydraulically connected to a hydraulic controlvalve 156, which receives pressurized hydraulic fluid from a hydraulicpump 158, which may be rotationally connected to the engine 134, anddirects such fluid to the lift cylinders, the tilt cylinder, the anglecylinders, and other hydraulic circuits or functions of the workmachine. The hydraulic control valve may meter such fluid out, orcontrol the flow rate of hydraulic fluid to each hydraulic circuit towhich it is connected. In alternative embodiments, the hydraulic controlvalve may not meter such fluid out but may instead only selectivelyprovide flow paths to these functions while metering is performed byanother component (e.g., a variable displacement hydraulic pump) or notperformed at all. The hydraulic control valve may meter such fluid outthrough a plurality of spools, whose positions control the flow ofhydraulic fluid, and other hydraulic logic. The spools may be actuatedby solenoids, pilots (e.g., pressurized hydraulic fluid acting on thespool), the pressure upstream or downstream of the spool, or somecombination of these and other elements.

In various embodiments, the controller 138 may send commands to actuatework implements 130, 162 in a number of different manners. As oneexample, the controller 138 may be in communication with a valvecontroller via a controlled area network (CAN) and may send commandsignals to the valve controller in the form of CAN messages. The valvecontroller may receive these messages from the controller and sendcurrent to specific solenoids within the electrohydraulic pilot valve160 based on those messages. As another example, the controller mayactuate a work implement 130, 162 by actuating an input in the operatorcab 136. For example, an operator may use a joystick to issue commandsto actuate the blade 130, and the joystick may generate hydraulicpressure signals, pilots, which are communicated to the hydrauliccontrol valve 156 to cause the actuation of the blade. In such aconfiguration, the controller may be in communication with electricaldevices (e.g., solenoids, motors) which may actuate a joystick in theoperator cab. In this way, the controller may actuate the blade byactuating these electrical devices instead of communicating signals toelectrohydraulic pilot valve.

As referenced above, the controller 138 is configured to receive inputsignals from some or all of various sensors 170 which may include imagedata sources such as cameras and collectively defining an imagingsystem. The sensors 170 may include video cameras configured to recordan original image stream and transmit corresponding data to thecontroller 138. In the alternative or in addition, the sensors 170 mayinclude one or more of an infrared camera, a stereoscopic camera, a PMDcamera, high resolution light detection and ranging (LiDAR) scanners,radar detectors, laser scanners, and the like within the scope of thepresent disclosure. The number and orientation of said sensors 170 mayvary in accordance with the type of work machine 100 and relevantapplications, but may at least be provided with respect to a field ofview 172 rearward of the work machine 100 and configured to captureimage data associated with surroundings including for example therear-mounted work implement 162 and other objects proximate thereto.

The position and size of an image region recorded by a respective cameraas a sensor 170 may depend on the arrangement and orientation of thecamera and the camera lens system, in particular the focal length of thelens of the camera. One of skill in the art may further appreciate thatimage data processing functions may be performed discretely at a givenimage data source if properly configured, but also or otherwise maygenerally include at least some image data processing by the controlleror other downstream data processor. For example, image data from any oneor more image data sources may be provided for three-dimensional pointcloud generation, image segmentation, object delineation andclassification, and the like, using image data processing tools as areknown in the art in combination with the objectives disclosed.

The controller 138 of the work machine 100 may be configured to produceoutputs, as further described below, to a user interface 142 associatedwith a display unit for display to the human operator. The controller138 may be configured to receive inputs from the user interface 142,such as user input provided via the user interface 142. Not specificallyrepresented in FIG. 2 , the controller 138 of the work machine 100 mayin some embodiments further receive inputs from and generate outputs toremote devices associated with a user via a respective user interface,for example a display unit with touchscreen interface. Data transmissionbetween for example the vehicle control system and a remote userinterface may take the form of a wireless communications system andassociated components as are conventionally known in the art. In certainembodiments, a remote user interface and vehicle control systems forrespective work machines may be further coordinated or otherwiseinteract with a remote server or other computing device for theperformance of operations in a system as disclosed herein.

The controller 138 may in various embodiments, as part of the controlsystem of FIG. 2 and further in line with the above-referenceddisclosure, be functionally linked to a reading device 166 asconventionally known in the art such as for example an RFID device,barcode scanner, or the like for obtaining readable information. Thereading device 166 may be a discrete device, or in other embodiments mayinclude a data processing module in combination with image data orequivalent data captured by the sensor 170. For example, a workimplement 130, 162 within a field of view 172 of a camera as the sensor170 may have a barcode or equivalent tags (e.g., AprilTags) 166associated with machine readable information, which may as furtherdescribed herein be used to identify and/or retrieve informationassociated with the work implement.

In an embodiment as shown, the controller 138 may further befunctionally linked to a work machine movement control system 168,wherein for example the controller may directly or indirectly generateoutput signals for controlling the steering and/or advance speed of thework machine 100. The controller 138 may alternatively or in additionreceive input signals from the movement control system 168 indicative ofthe steering and/or advance speed of the work machine 100.

An embodiment of a method 400 of the present disclosure may now bedescribed with further illustrative reference to FIGS. 3-7 . The presentembodiment is intended as illustrative and the associated description isnot limiting on the scope of any other embodiments unless otherwisespecifically noted herein. It should also be noted that various steps asdisclosed in accordance with the present embodiment may be combined,omitted, or supplemented by one of skill in the art when considering theapplicable functions and without necessarily altering the scope of thepresent disclosure, unless otherwise expressly provided herein.

The exemplary method 400 as illustrated begins in step 410 by capturingdata (e.g., image data) in at least one field of view 172. As furtherdescribed below, the relevant field of view is associated with arear-mounted camera as a sensor 170 and comprises surroundings at leastto the rear of the work machine 100.

In step 420, the system (via the controller 138 directly, or indirectlyvia one or more associated image processing components or modules)classifies objects in respective locations in the field of view 172, andfurther determines from the captured data whether a portion of the workmachine 100 or an attachment thereto, such as for example rear-mountedwork implement 162, is present in the field of view 172.

Such a determination may be accomplished in numerous ways within thescope of the present disclosure. For example, in step 430 as shown, thesystem obtains information regarding for example physical contours ofthe work implement 162 and generates a bounding region 174 about saidcontours. The bounding region 174 may be a relatively simplethree-dimensional polyhedron as illustrated in the figures, or in otherembodiments may include any number of sides, angles, curved faces, orthe like as better or more precisely corresponding with the actualcontours of the work implement 162. In certain embodiments wherein theperspective is substantially horizontal or substantially vertical inorientation (e.g., a bird's eye view) the bounding region 174 and theassociated image segmentation mask 176 as further described below may bea relatively simple two-dimensional polygon, or may include any numberof sides, angles, curved faces, or the like as better or more preciselycorresponding with the actual contours of the work implement 162.

In various embodiments, the information regarding a presence and/orphysical contours of the work implement 162 may be obtained in one ormore of numerous forms. In one example as referenced above (step 412),machine readable tags such as for example AprilTags, RFID tags and thelike may be provided on the work implement 162 itself, such thatscanning of the tags enables simple retrieval of the associatedinformation (e.g., a type of work implement or more specific informationregarding the unique implement itself) by the controller 138. In anotherexample (step 414), conventional image classification techniques may beutilized to recognize the presence and/or at least roughly determinecontours of the work implement 162. For example, the system may beconfigured to ascertain a work implement 162 or type of work implementthat is coupled to the work machine 100 by comparing a detected shape toa library of work implement shapes, wherein an appropriate imagesegmentation mask may be selected. In another example (step 416),movement and/or position sensors 164 associated with the work implement162 may generate output signals to the controller 138 which, taken aloneor further in combination with one or more other sensors 132, 144 may beindicative of a position of the rear-mounted implement 162 or alikelihood that the implement 162 is in the relevant field of view 172.

In step 440, the system applies at least one image segmentation mask 176in association with the generated bounding region 174, and independentlyof any objects determined to be separate from portions of the workmachine 100 or attachments (e.g., work implements 162) thereto andextending into the field of view 172. In step 442, the system generatesa display on the display unit 142 in the operator cab, or in someembodiments a remote display unit, with a masked area corresponding tothe image segmentation mask 176. Such a display may comprise images fromthe same rear-view perspective as an orientation of the associatedsensor 170, or in various embodiments may comprise a top-down (e.g.,bird's eye) view, surround view, or other views as may be for examplegenerated by stitching together images from a plurality of sensors 170.Where a number of views are available, user selection from among suchviews may optionally be enabled via the user interface. In anembodiment, the masked area may not be simply obscured but ratheraugmentation may be utilized to generate an image corresponding toobjects in an area behind the masked area, for example using additionalcameras that are not obscured in the relevant area by the same workimplement 162. In step 444, the system performs an object detectionand/or recognition function utilizing the image segmentation mask 176,or otherwise stated an image segmentation mask may facilitate objectseparation such that objects other than the masked work implement 162may be reliably detected and/or recognized by the system.

In one example, with reference to a camera view from the upper rearportion of a work machine 100 as shown in FIG. 3A, a first object 310 isalongside the rear-mounted work implement 162 and a second object 320 isbehind the rear-mounted work implement 162. A digitized imageclassification may be unable to reliably distinguish the first object310, second object 320, and work implement 162, as represented forexample in FIG. 3B. A depth disparity view according to conventionalimage processing techniques may indicate that the second object 320 isnot at the same depth as the work implement 162, but would typically beunable to distinguish the first object 310 and the work implement 162 asbeing at the same depth. With an appropriate bounding region 174 and anassociated image segmentation mask 176 applied to the work implement162, however, the work implement 162 may be ignored and the variousobjects 310, 320 reliably separated by the image processing system forthe purpose of determining their respective identities, locations,movements, or the like.

In various embodiments, the form of the bounding region 174 and theassociated image segmentation mask 176 may substantially correlate withthe contours of the work implement 162 itself, for example wheredetailed information regarding said contours is available for retrievalupon identifying the particular work implement and/or currentorientation thereof using machine readable tags or the like. In otherembodiments, the form of the bounding region 174 and/or the imagesegmentation mask 176 may be more generalized. A more detailed boundingregion 174 and image segmentation mask 176 may typically be preferred,as may be illustrated for example by reference to FIGS. 4A, 4B, 5A, and5B.

As represented in FIG. 4A, an object 310 (a human in the presentexample) is standing in the field of view 172 of a rear-mounted cameraas sensor 170, behind the rear-mounted work implement 162. A boundingregion 174 about the work implement 162 in the present example does notneed to be highly precise, as further illustrated by reference to FIG.4B. The corresponding image segmentation mask 176 may be applied toenable separation of the object 310 from the work implement 162 in astraightforward manner. As represented in FIG. 5A, however, the object310 is still in the field of view 172 of the camera as sensor 170 but isnow present between the work implement 162 and the main frame 140 of thework machine 100. If the image segmentation mask 176 were to be appliedbroadly with respect to a simple geometrically defined bounding region174, appropriate separation of the object 310 may not be possible.Accordingly, a more precisely defined bounding region 174 and imagesegmentation mask 176 with respect to the contours of the respectivework implement 162 may be preferable for at least this context, whereinan object 310 in close proximity to the work machine 100 may be reliablyseparated and identified by elements of the system.

In step 450, the system may be configured to generate any alerts thatmay be triggered by or otherwise appropriate in view of the objectdetection and/or recognition function. For example, if an object isrecognized as being in a dangerous location relative to a position ofthe work implement, or otherwise in view of a predicted movement of thework machine and/or work implement, an alert may be generated in theform of an audio alarm, a visual alert on the display unit, or the like.

The type of alert may be dependent at least in part on the type ofobject, wherein for example a living creature as a first type ofdifferentiated object in the field of view may result in a first andmore urgent form of alert whereas a second type of differentiated objectin the form of for example debris may result in a second and less urgentform of alert. In some embodiments this determination may be made inview of the above-referenced object detection and/or recognitionfunction, further in view of a machine geometry or pose detectionfunction which determines if the work implement 162 is in a position ororientation corresponding with for example a first work state (i.e., atrest and therefore of reduced risk to a proximate object) or a secondwork state (i.e., an active state and therefore of potentiallyheightened risk to a proximate object).

An alert according to step 450 may be generated visually for the benefitof the operator, and/or may include output signals generated to preventactuation of the work implement 162 or other elements of the workmachine 100 that could cause a collision or other intersection with adetected and/or recognized object.

Alert functions may be generated in certain embodiments in associationwith a predetermined threshold for a given work implement 162, avariable threshold depending on a work state or condition of the givenwork implement 162, and/or a non-threshold determination made further inview of factors including for example a detected movement of theobject(s), detected movement of the work implement 162, predictedmovements of the object(s) and/or work implement 162, type of terrainbeing traversed by the work machine 100, orientation of the work machine100, and the like.

In some embodiments and further with reference to step 460, the systemmay be configured to determine whether the work implement 162 is movingor whether such movement is predicted in the field of view 172, and thenaccordingly to dynamically adjust application of the bounding region 174and the corresponding at least one image segmentation mask 176 so thatthe masked region persists with an actual position of the work implement162 in the field of view 172.

Movement of the work implement 162 may for example be determined and/orpredicted using the above-referenced inputs (step 416) fromimplement-mounted sensors 164, and/or using inputs received (step 418)from the work machine steering control system 168. Regardless of whetherthe rear-mounted work implement 162 is a ripper assembly whichsubstantially moves along with movement of the work machine frame 140,or a towed implement such as a tamping roller which is pivotally coupledabout a vertical axis to the work machine frame 140, movement of thework implement may reliably be predicted based on knowledge of the typeof attachment even if means are not available for directly sensingmovement relative to the field of view 172.

As used herein, the phrase “one or more of,” when used with a list ofitems, means that different combinations of one or more of the items maybe used and only one of each item in the list may be needed. Forexample, “one or more of” item A, item B, and item C may include, forexample, without limitation, item A or item A and item B. This examplealso may include item A, item B, and item C, or item B and item C.

Thus, it is seen that the apparatus and methods of the presentdisclosure readily achieve the ends and advantages mentioned as well asthose inherent therein. While certain preferred embodiments of thedisclosure have been illustrated and described for present purposes,numerous changes in the arrangement and construction of parts and stepsmay be made by those skilled in the art, which changes are encompassedwithin the scope and spirit of the present disclosure as defined by theappended claims. Each disclosed feature or embodiment may be combinedwith any of the other disclosed features or embodiments.

What is claimed is:
 1. A method of object detection for a work machinecomprising a main frame, the method comprising: receiving, from at leastone sensor associated with the work machine, data corresponding to afield of view extending from the main frame; classifying objects inrespective locations in the field of view; generating at least onesegmentation mask corresponding to contours for at least one portion of,or attachment to, the work machine as determined to be in the field ofview, wherein each of the at least one segmentation mask defines arespective masked zone in the field of view; and determining one or moreof the classified objects to be separate from the at least one portionof, or attachment to, the work machine, wherein the at least onesegmentation mask is applied to the at least one portion of, orattachment to, the work machine independently of any of the one or moreseparate objects in the field of view.
 2. The method of claim 1, furthercomprising generating images on a display unit corresponding to thefield of view and having the at least one segmentation mask appliedthereto.
 3. The method of claim 1, comprising: during movement of thework machine, detecting from the received data at least one staticportion in the field of view and one or more dynamic portions relativethereto in the field of view; and generating the at least onesegmentation mask corresponding to the detected at least one staticportion in the field of view.
 4. The method of claim 1, wherein datacorresponding to at least one portion of, and/or attachment to, the workmachine is retrieved and applied for generation of an associatedsegmentation mask based on user input.
 5. The method of claim 4, whereinthe user input comprises a user selection from among a library ofselectable portions of and/or attachments to the work machine.
 6. Themethod of claim 1, further comprising dynamically generating the atleast one segmentation mask based on determined movements of anattachment to the work machine relative to the field of view.
 7. Themethod of claim 6, wherein the movements of the attachment aredetermined based on detected steering signals for the work machine. 8.The method of claim 6, wherein the movements of the attachment aredetermined based on first input signals from a sensor associated withthe attachment and second input signals from a sensor associated withthe work machine.
 9. The method of claim 1, wherein a bounding regionfor the at least one portion of, or attachment to, the work machine isdetermined via image classification from the received data.
 10. Themethod of claim 1, wherein a bounding region for the at least oneportion of, or attachment to, the work machine is determined at least inpart via output signals from one or more movement sensors associatedwith the respective portion of, or attachment to, the work machine. 11.The method of claim 1, wherein a bounding region for the at least oneportion of, or attachment to, the work machine is determined by scanninga machine readable tag associated with the respective at least oneportion of, or attachment to, the work machine and retrieving boundingregion data corresponding to the respective at least one portion of, orattachment to, the work machine from data storage based on the scannedmachine readable tag.
 12. The method of claim 1, comprising: detectingrespective positions of the one or more further objects determined to bein the field of view, relative to the work machine and/or the at leastone portion of, or attachment to, the work machine, and conditionallygenerating output signals corresponding to at least an identified unsafeposition of an object relative to at least one of the work machineand/or the at least one portion of, or attachment to, the work machine.13. A work machine comprising: a main frame supported by one or moreground engaging units; at least one sensor configured to generate datacorresponding to a field of view extending from the main frame; and acontroller linked to receive the data from the at least one sensor andconfigured to classify objects in respective locations in the field ofview, generate at least one segmentation mask corresponding to contoursfor at least one portion of, or attachment to, the work machine asdetermined to be in the field of view, wherein each of the at least onesegmentation mask defines a respective masked zone in the field of view,and determine one or more of the classified objects to be separate fromthe at least one portion of, or attachment to, the work machine, whereinthe at least one segmentation mask is applied to the at least oneportion of, or attachment to, the work machine independently of any ofthe one or more separate objects in the field of view.
 14. The workmachine of claim 13, wherein the controller is configured to: duringmovement of the work machine, detect from the received data at least onestatic portion in the field of view and one or more dynamic portionsrelative thereto in the field of view; and generate the at least onesegmentation mask corresponding to the detected at least one staticportion in the field of view.
 15. The work machine of claim 13, whereindata corresponding to at least one portion of, and/or attachment to, thework machine is retrieved from data storage and applied for generationof an associated segmentation mask based on user input via a userinterface linked to the controller.
 16. The work machine of claim 15,wherein the user input comprises a user selection from among a libraryof selectable portions of and/or attachments to the work machine. 17.The work machine of claim 13, wherein the controller is furtherconfigured to generate images for display on a display unitcorresponding to the field of view and having the at least onesegmentation mask applied thereto.
 18. The work machine of claim 13,wherein the controller is further configured to dynamically generate theat least one segmentation mask based on determined movements of anattachment to the work machine relative to the field of view.
 19. Thework machine of claim 18, wherein the movements of the attachment aredetermined based on detected steering signals for the work machine. 20.The work machine of claim 18, wherein the movements of the attachmentare determined based on first input signals from a sensor associatedwith the attachment and second input signals from a sensor associatedwith the work machine.